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The Big Improvement in PISA 2009 Reading 
Achievements in Serbia: Improvement of the Quality of 
Education or Something Else? 
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The PISA 2009 results in Serbia show a big improvement in reading lit¬ 
eracy compared to 2006 - the average score is 41 points higher, which 
is equal to the effect of a whole year of schooling in OECD countries 
and represents the second highest improvement ever recorded in a PISA 
study. In the present paper, we discuss potential reasons for such a big 
improvement based on analysis of the PISA 2009 reading achievements 
in different countries, with a special focus on countries from the same 
region (Croatia, Slovenia, Montenegro, Bulgaria, Romania and Albania). 
The analysis shows that the largest part of the improvement was realised 
at lower achieving levels, suggesting that the dominant method of teach¬ 
ing in schools is a traditional method oriented towards the acquisition 
and reproduction of academic knowledge. Findings of data analysis sup¬ 
port the conclusion that the improvement is mainly the result of certain 
contextual factors, such as higher student motivation and a high level of 
official support for the PISA study in Serbia, rather than representing a 
real improvement in the quality of education. 
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Reading as an educational outcome: operationalisation 
and possible approaches to measurement 

Measurement of reading as an important educational outcome has a 
tradition as long as the testing of knowledge itself, having been part of school 
and research practice since the beginning of the twentieth century. In parallel 
with changes in society, the economy and culture, as well as with more inten¬ 
sive research into the nature of reading processes, the definition of reading has 
changed. Consequently, the instruments used to measure this concept have also 
changed and developed. 

In the middle of the last century, under the influence of behaviour¬ 
ism and later of information processing theory, the belief that reading com¬ 
prehension ability is a series of discrete mental abilities that form a hierarchy 
was predominant. However, the research focuses of these two approaches were 
different: behaviourists were more involved in the structural aspects of read¬ 
ing (trying to answer the question as to which activities participate in read¬ 
ing), while the cognitive approach dealt with the functional aspects of reading 
processes (how activities are coordinated). Thanks to research that originated 
under the influence of ideas from information processing theory (LaBerge & 
Samuels, 1974), we gained a plausible explanation of the role of attention in 
fluent reading, as well as a definition of the process of text understanding as 
a structure of meanings built through a series of hypotheses that the reader 
formulates, verifies, adopts or discards while reading. For these researchers, 
reading is “uncertainty” that exists in the reader, not on the page, and hypoth¬ 
eses are formulated in order to reduce this uncertainty. Efficiency in reading 
increases with the number of confirmed hypotheses and with more sub-skills 
being brought to the level of automation. 

Understanding educational outcomes, including reading, in line with 
these conceptions means that knowledge and skills can be broken down into 
components, with each component always behaving in the same way, regardless 
of contextual factors. In this case, in the construction of test sets there is a strict 
requirement that tests should be one-dimensional, i.e., that all items in a sub¬ 
test measure the same characteristic, ensuring that the score has essentially the 
same meaning for all individuals (the assumption of universality). Reading is 
seen as the sum of discrete abilities and skills (e.g., text understanding, reading 
speed, oral and written production) that are taught gradually, and their relative 
contribution can be determined by factor analysis. Tests developed within the 
framework of this approach are mainly composed of multiple choice questions, 
suggesting that the main task of the respondents is to find the correct answer 
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and not to engage in interpretation. 

Since the 1970s, measuring reading has been impacted by the penetrat¬ 
ing socio-cultural theory of Vygotsky on the interdependence of thought and 
language, and since the 1980s, when the importance of metacognition was rec¬ 
ognised in cognitive psychology, researchers of reading have been oriented to¬ 
wards identifying metacognitive strategies that facilitate and control awareness 
of the level of text understanding. Metacognitive strategies were identified by 
comparing reading behaviour and introspective reports about reading given by 
good and poor readers. For example, in one study (Brown, 1980) good readers 
were asked to describe how clear the goals of reading are and to identify impor¬ 
tant aspects of text messages. They were then asked how they control whether 
they understand the text and whether they are taking the correct steps. Ac¬ 
cording to Underwood, when formative assessment and metacognitive reading 
comprehension strategies that facilitate understanding entered the reading area 
in 1970s and 1980s, our understanding of the nature of reading and the nature 
of training in reading were changed forever (Underwood, 1997). 

The beginning of 21 st century was marked by a rapid increase in the num¬ 
ber of studies of educational achievement, caused by the change of the existing 
conceptual paradigm and clearly visible in the expansion of research studies 
that have largely abandoned the traditional principles of psychometric testing. 

What caused this paradigm shift? One reason (although not the most 
important reason) for the change in conceptual approach is the increased theo¬ 
retical knowledge (and, based on this, empirical knowledge) about the nature 
of learning processes that was developed within constructivist and cognitive 
theories of psychological development. It would be more accurate to say that 
rather than opening up the issue of the quality assessment of educational out¬ 
comes, theoretical knowledge has provided answers. Some of the issues and 
concerns arising from the practical application of findings derived from the 
external evaluation of educational achievement in order to improve teaching 
practices are: how could assessment results and the objectives and functions of 
education be harmonised, and does education justify the investment? 

An analysis of the theoretical approaches that have shaped the concep¬ 
tual framework of the modern assessment of educational outcomes, especially 
the PISA project, shows that the term “competence” is frequently used as a cen¬ 
tral construct. Competences are defined and operationalised within “concep¬ 
tual pragmatism,” which defines this construct in a pragmatically relevant and 
scientifically plausible way (relying primarily on theoretical knowledge about 
the nature and structural characteristics of the knowledge developed within 
social constructivism and cognitive orientations). 
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One of the definitions of competences developed in this approach was 
adopted in the OECD project DeSeCo (Definition and Selection of Competen¬ 
cies), with which the PISA research study abuts directly. Competency is defined 
as the capacity of an individual to successfully respond to complex, composite 
requirements in a particular context through the mobilisation of psychoso¬ 
cial conditions, including their cognitive and non-cognitive aspects (Rychen 
& Salganik, 2003). In other words, competencies are seen as internal mental 
structures, as dispositions, or resources “embedded” in the individual. There is 
a wide range of attributes that are seen as components of the internal structure 
of competence. There is no disagreement amongst various authors about the 
fact that both higher-order cognitive skills (e.g., analytical or critical thinking, 
decision-making ability, problem-solving ability) and total or specific knowl¬ 
edge must be mobilised for the attainment of competent achievement (Kirsch 
et al., 2002; Rychen & Salganik, 2003). 

The concept of reading literacy adopted in PISA relies on cognitive con¬ 
cepts that highlight the interactive nature of the reading process and the crea¬ 
tive nature of the process of understanding. Cognitive scientists argue that the 
meaning of the text is constructed in the interaction between the text and the 
reader (Underwood, 2007). In this interaction, the reader brings cognitive and 
metacognitive strategies to work on the text, as well as previous knowledge and 
experience, including specific knowledge and experience in reading situations, 
such as the use of textual and situational incentives. The text contains certain 
linguistic and structural elements and addresses a specific topic, while context 
determines the purpose of reading and the selection of reading strategies ap¬ 
propriate to the text. 

Reading literacy as an educational competence 

The definition of reading literacy in the PISA study, similar to that in 
other international reading assessments, such as PIRLS (Progress in Interna¬ 
tional Reading Literacy Study, in Mullis et al., 2007) and ALL (Adult Literacy 
and Lifeskills, in Lemke & Gonzales, 2006), emphasises the importance of 
reading in active and critical participation in society, thereby promoting the 
ability of students to read and to critically analyse information and use it for 
different purposes. Changes in the definition of the learning process and pro¬ 
motion of the concept of lifelong education have led to a broader understand¬ 
ing of the concept of literacy. Literacy does not only mean the skill of decoding 
written words and literal understanding of the meaning, which is typically mas¬ 
tered during the first years of schooling; literacy means merging functional and 
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transfer knowledge, skills and strategies that people acquire throughout their 
lives and through interaction in the social groups to which they belong. Es¬ 
sentially, this definition of literacy says that reading is not a unitary skill, but 
rather a compilation of processes, approaches and skills that vary depending on 
the reader and the type of text, as well as the goal or situation in which the text 
is read (Campbell et al., 2001). Based on these definitions, the PISA project has 
developed the following definition of reading literacy: understanding, using and 
reflecting on written texts in order to achieve personal goals, develop skills and 
potential, and to contribute to community life (Kirsh et al, 2002; OECD, 2010). 

In other words, the concept of literacy describes the capacity of students 
to apply knowledge and skills in real life situations, and to analyse, draw con¬ 
clusions and accurately communicate the solutions arrived at. 

This definition implies a broad range of situations in which reading lit¬ 
eracy plays an important role, ranging from an individual’s aspirations, acquir¬ 
ing qualifications or finding a job, to less specific situations, such as meeting the 
challenges of modern society in order to enrich and improve one’s quality of 
life. In accordance with the different contexts in which reading takes place, the 
assessment of reading literacy involves using a range of different types of texts. 

Reading Research in Serbia 

Assessing reading as an educational outcome does not have a long tra¬ 
dition in Serbia. The practice of testing achievements in the area of reading is 
closely related to the very few assessment studies in education. In spite of the 
fact that reading was defined and operationalised as an educational outcome 
of great importance in these research studies (and, therefore, reading achieve¬ 
ment represents a measure of the quality of the education system), research 
findings have not significantly influenced the educational policy, if at all (Bau- 
cal & Pavlovic Babic, 2010). In the curriculum, reading is reduced to a cor¬ 
respondence between written text and speech. On the other hand, in addition 
to conventional tests of knowledge in the area of language (spelling, grammar, 
knowledge of literature, vocabulary), the first comprehensive assessment of 
educational achievements (Havelka et al., 1990; UNICEF, 2001) also included 
the measurement of reading speed and reading comprehension. Since 2000, by 
participating in the PISA study and developing national assessment studies, the 
research focus has shifted from testing isolated language skills and knowledge 
to complex skills that are manifested through work on the text (Baucal et al., 
2007; Baucal & Pavlovic Babic, 2010; Pejic et al., 2009). Findings reported in the 
present paper belong to this research orientation. Operationalisation of reading 
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literacy and research design make the findings relevant for educational policies, 
and achievement in reading literacy is seen as one of the indicators of the qual¬ 
ity of education in Serbia. 

The main characteristics of reading literacy 
in the PISA project 

In the process of the operationalisation of reading literacy, it became 
evident that there were three main characteristics on which the determination 
of reading is based: the text, the aspects and the situations. Each of these key 
features was further developed into subcategories that serve for the further op¬ 
erationalisation of reading. 

Text. Until recently, a correct definition of reading would include texts 
written (printed) on paper. Today, it is a common, everyday activity to read a 
text on a screen. In the PISA 2009 study, Serbia did not participate in testing 
reading literacy in digital texts. Students from Serbia worked on texts that were, 
in terms of format, linear (continuous) or nonlinear (not continuous) and had a 
sequential organisation, thus demanding different approaches of the reader. In 
terms of type of presentation and content, texts are classified into the following 
categories: description (information related to the characteristics of an object 
- typically answers to What questions), narration (text responding to the ques¬ 
tions such as When, and In what order), presentation (answers to How ques¬ 
tions), argumentation (arguments and proposals are exposed, often answers to 
Why questions), instructions (instructions on how to do something), and ex¬ 
change (text interacting with readers and exchanging information with them). 

Aspects. These are, in fact, mental strategies, approaches and intentions 
used by the reader, classified into three main categories: access to information 
and information retrieval, integration and interpretation, reflection and evalua¬ 
tion. Access to information and information retrieval means browsing, search¬ 
ing, and the identification and selection of relevant information - the retrieval 
of information assumed relevant or the automatic understanding of the text. 
There is little or no interpretation. There are no gaps within the meaning of the 
text that need to be compensated for - meaning is evident and clearly stated 
in the text. The reader must recognise the importance of information or ideas. 
Integration and interpretation are processes that we use to build the meaning 
of the text. Integration refers to the establishment of a relationship (or relation¬ 
ships) between parts of the text. Interpretation refers to the process of build¬ 
ing sense, based on information that is not (always) complete or explicit. It 
also involves developing and deepening first impressions, as well as acquiring 
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a more specific and more complete understanding of the text. While building 
the meaning of the text, the reader also formulates conclusions on information 
or ideas that are not explicitly given. This allows the reader to draw conclu¬ 
sions that go beyond literal interpretation of the text and to fill in gaps and 
uncertainties. For successful readers, these processes are brought to the level of 
automation. Reflection and evaluation - thinking about the text and evaluat¬ 
ing its content or form of interaction - implies a reference to prior knowledge, 
experiences and ideas. The reader compares the facts and opinions expressed 
in the text with his/her own knowledge and opinions, assesses their founda¬ 
tion, reveals contradictions and inconsistencies, analysing arguments, evidence 
and refutations, and finally articulates and expounds his/her conclusions and 
attitudes. He/she looks for evidence in the text and pits it against evidence from 
other sources of information, using general and specific knowledge, but also 
the ability of abstract thinking. 

Situation. Based on content, purpose of reading and the students’ rela¬ 
tionship with the context to which the text refers, texts are classified into four 
categories: personal, educational, occupational and public. 

Method 

Research Design 

The PISA study uses the balanced incomplete block (BIB) assessment 
design (Johnson, 1992; NAEP, 2001). The BIB design has been developed for 
large-scale assessments in order to enable measurement of a broad range of 
competencies or knowledge, while limiting the time of participants engage¬ 
ment to 2-3 hours. In order to obtain reliable individual measures of different 
competences, a relatively large number of items needs to be used. So as to re¬ 
duce each participant’s time of engagement, items are organised into a number 
of item blocks, which are connected according to a specific scheme into book¬ 
lets. Thus each booklet contains only a part of the items, chosen in such way 
that the content in each booklet overlaps with two other brochures. The BIB 
design requires the use of IRT techniques to analyse the data (Birnbaum, 1968; 
Bond & Fox, 2007; Lord, 1980). 

Sample 

The PISA sample targets 15-year-old students, regardless of the class 
that they are attending at the time. In Serbia, the PISA 2009 sample was strati¬ 
fied - schools are the first stratum and students the second. The PISA 2009 
study in Serbia involved 190 schools, mostly upper secondary schools. In each 
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secondary school, 35 students were selected (fewer students were selected in 
primary schools, where there is a small proportion of r5-year-old students). The 
student sample in Serbia was also designed to be representative of the type of 
educational programme in upper secondary education. The planned sample 
size was 5804, of which a total of 5523 students were tested (about 95% of the 
planned sample). The structure of the sample by gender and the class attended 
by the student at the time of PISA assessment is shown in Table r. 

Table 1: Structure of the sample of students from Serbia who were tested in the 
PISA 2009 study according to gender and class attended at the time of PISA 
assessment 




Females 


Males 




Freq. 

% 

Freq. 

% 


7 th grade 

2 

0.1% 

4 

0.1% 

Compulsory primary education 

8 th grade 

21 

0.7% 

38 

1.4% 

Upper secondary education 

1 st grade 

2757 

97.0% 

2594 

96.8% 

2 nd grade 

63 

2.2% 

44 

1.6% 


Total 

2843 

100.0% 

2680 

100.0% 


Instrument 

The instrument for the assessment of reading literacy in the PISA 2009 
study consisted of r3 brochures that contained a total of i3r items. The brochures 
were distributed to the students according to the spiral method (NAEP, 2001) 
so that between 4r6 and 439 students were surveyed by each of the r3 booklets. 

Each item was designed to examine one component of reading lit¬ 
eracy and particular types of texts. In addition, each item was contextu¬ 
alised so that it applied to a personal, social, professional or educational 
context. In other words, each item can be described by three dimensions: 
component of reading, type of text and type of context. Items also var¬ 
ied according to formal characteristics: closed items, complex closed 
items, limited open items, open items and items with a short answer. 

Results 

General achievements on the PISA reading scale 

To what extent was reading competence developed in the r5-year-olds from 
different countries? Based on the data, it is possible to generate different indica¬ 
tors of student achievements. In the present paper, the average achievement, per¬ 
centiles, distribution by levels of achievement and data on trends are analysed. 
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Figure 1 shows the average achievements on the PISA reading literacy 
scale of students from different countries in Europe. In addition to the average 
achievement, the data indicates the extent to which the achievements of stu¬ 
dents within each country differ. The differences that exist among students are 
described by percentiles (10%, 25%, 75%, and 90%). 

The data show that students from Finland have the highest score 
(M=536, SE=2-3), which is 43 points higher than the OECD average (M=493, 
SE=o.5). Bearing in mind that one year of schooling, according to estimates for 
OECD countries, has an average effect of about 40 points (OECD, 2010), we can 
say that the education system in Finland supports the development of reading 
literacy to a greater extent than education systems in other OECD countries, as 
the difference corresponds to the effect of one year of schooling. In other words, 
when the reading literacy of students from Finland is compared to the reading 
literacy of students from other OECD countries one could gain the impression 
that the students have been educated one year longer. Since there is no signifi¬ 
cant difference in the number of years students spend in the education system 
up to the age of 15, this means that the education system in Finland is more 
effective than those of other OECD countries. In addition to students from 
Finland, students from the Netherlands (M=5o8), Belgium (M=5o6), Norway 
(M=503), Estonia (M=50i), Switzerland (M=50i), Poland (M=50o), Iceland 
(M=5oo) and Lichtenstein (M=499) also attained average achievement that is 
statistically significantly higher than the OECD average. 
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Finland (536) 
The Netherlands (508) 
Belgium (506) 
Norway (503) 
Estonia (501) 
Switzerland(501) 
Poland(500) 
Island (500) 
Liechenstein(499) 
Sweden (497) 
Germany(497) 
Ireland (496) 
France (496) 
Denmark (495) 
UK (494) 
Hungary (494) 
OECD average (493) 
Portugal (489) 
Italy (486) 
Latvia (484) 
Slovenia (483) 
Greece (483) 
Spain (481) 
Czech Republic (478) 
Slovakia (477) 
Croatia (476) 
Luxembourg (472) 
Austrija (470) 
Lithuania (468) 
Russia (459) 
Serbia (442) 
Bulgaria (429) 
Romania (424) 
Montenegro (408) 
Albania (385) 
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Figure 1: Average student achievements on the PISA reading scale, and 
student achievements on 10, 25, 75, and 90 percentile, (data for selected 
countries in Europe) 

The average achievement on the reading scale of students from Serbia 
was about 442 points (SE=2-4). This is statistically significantly lower than the 
OECD average, with a difference is about 50 points, which corresponds to the 
effect of about 1.25 years of schooling in OECD countries. This suggests that the 
education system in Serbia is somewhat “less supportive” of the development of 
PISA reading competence compared to OECD countries. 

Compared with students from other countries in the region (Table 2), 
students from Serbia have a similar level of reading literacy to students from 
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Bulgaria, and a higher level than students from Albania (M=385), Montenegro 
(M=4 o 8) and Romania (M=424), while the level of reading literacy of students 
from Serbia is significantly lower than students from Croatia (M=476) and Slo¬ 
venia (M=483). The average achievement of students from Serbia is about 30-40 
points lower than the average achievement of students educated in Croatia and 
Slovenia, which corresponds to the effect of almost one year of schooling. 

When the trend in the reading literacy of students from Serbia is ana¬ 
lysed, it can be seen that a big improvement was achieved between the PISA 
2006 and 2009 studies (Table 2). In the PISA 2003 study, students from Serbia 
achieved an average of 412 points (OECD, 2004). In the next cycle, in 2006, 
the reading literacy of students from Serbia dropped by 11 points to 401 points 
(OECD, 2007). In the PISA 2009 study, however, the average achievement of 
students from Serbia was 41 points higher than in 2006. This improvement is 
similar to the effect of one year of schooling in OECD countries and is one of 
biggest improvements ever recorded in a PISA study. The average achievement 
of students from Montenegro, Bulgaria and Romania also improved between 
these two PISA cycles, by between 16 and 28 points. In the same period, the 
average achievement of students from Croatia remained at the same level as in 
2006, while the average achievement of students from Slovenia decreased by 11 
points. 


Table 2: Average student achievement on the reading literacy scale in 2003, 
2006, and 2009. 


Country 

2003 

2006 

2009 

Difference between 2009 
and 2006 

Serbia 

412 

401 

442 

+41 

Croatia 

- 

477 

476 

-1 

Slovenia 

- 

494 

483 

-11 

Montenegro 

- 

392 

408 

+16 

Bulgaria 

- 

402 

429 

+27 

Romania 

- 

396 

424 

+28 

Albania 

- 

- 

385 

- 


Internal differences in student achievement within specific 
countries 

In addition to the average achievement, it is important to take into ac¬ 
count the differences that exist between students within each country. Here the 
focus will be on differences that exist in Serbia. As can be seen from the data 
shown in Figure 1, the lower quartile of students in Serbia achieved fewer than 
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388 points, which places them amongst the very low achievers, while the upper 
quartile of the most successful students achieved above 500 points. In addition, 
the lower deciles of students from Serbia fall below 330 points, while the 10% of 
students with the highest scores achieved scores above 547 points. 

Although these differences in student achievement from the lower/up¬ 
per quartiles and deciles are rather large, they are lower than those from other 
countries. For example, although the average achievement of students from 
Serbia and Bulgaria is at a similar level, the differences between students from 
Bulgaria are much larger than the differences between students being educated 
in Serbia (this is illustrated by the length of the bar for Bulgaria and Serbia in 
Figure 1). The inter-quartile difference in Bulgaria is about 160 points, while 
that of Serbia is considerably lower - about 110 points. As a result of this, the 
top 10% of students from Bulgaria have scores that are significantly higher than 
the average top 10% score of students from Serbia (572 vs. 547 points). Flowever, 
when achievements of the 10% of students with the lowest scores are compared, 
the opposite situation is evident - low performing students from Serbia have 
significantly higher achievements (331 vs. 276 points). 

The development of individual components of reading literacy 

Since reading literacy was the central domain in the PISA 2009 assess¬ 
ment, data for different components of reading competence were also provided. 
In Table 3, data on student achievements in different components of reading 
literacy are presented. The data are expressed as the difference between the av¬ 
erage achievement of students in certain components and the average achieve¬ 
ment on the reading scale. If the difference is positive, this suggests that students 
are somewhat more successful in a given component, while a negative differ¬ 
ence means that students are less successful in this aspect of reading compe¬ 
tence. Thus the profile of the achievement of students from different countries 
in different components may indicate relative advantages and shortcomings of 
the respective education system in terms of providing learning opportunities to 
students to develop their reading competence (OECD, 2010). 

The data from Table 3 suggest that students from Serbia were relatively 
successful in identifying and selecting information in the text, while they were 
significantly less successful in reflecting on and evaluating the texts. Accord¬ 
ing to this profile, students from Serbia are most similar to students from the 
following European countries: Slovenia, Austria, Slovakia and Hungary. Stu¬ 
dents from the UK and Greece have the opposite profile - these students are 
relatively more successful in terms of reflecting on and evaluating the informa¬ 
tion and texts than in terms of identifying and selecting information. Students 



C-E-P-S Journal I Vol.l [N“3| Year 2011 


from Serbia deal better with linear texts than with nonlinear texts, following 
the general trend in other countries. This means that students from Serbia are 
more successful in working with traditional texts that present information in 
context, while they are somewhat less successful with nonlinear texts (such as, 
for example, graphs, tables, diagrams, maps, forms, advertisements, etc.). 


Table 3: Comparison of average achievements in different components of 
reading literacy in relation to the average student score on the reading scale. 


The difference between the average achievement of certain compo¬ 
nents and the average achievement on the reading literacy scale 



Average 

score 


Reading aspects 


Different types of texts 


Approach 
and finding 

Connecting 
and inter¬ 
preting 

Processing 
and inter¬ 
preting 

Linear texts 

Nonlinear 

texts 

Serbia 

442 

7 

3 

-12 

2 

-4 

OECD 

493 

2 

0 

1 

0 

0 


Different levels of reading literacy: the distribution of students by 
levels 

In addition to the average achievements of students, the achievements of 
students are also described by the percentage of students who attained each of 
the six levels of achievement - from the lowest level (level 1) to the highest level 
(level 6). Level 2 has a special significance, since it is treated as the threshold of 
functional reading literacy both in the PISA study and in EU statistics. Students 
at this level can understand and cope with only simple, familiar texts in which 
the important information is clearly marked and easily distinguishable. Exist¬ 
ing studies show that young people who are below level 2 at the age of 15 will be 
faced with significant difficulties in terms of future education and employment 
opportunities (Bertschy, Cattaneo, & Wolter, 2008). 

Figure 2 shows data on the percentage of students who are below level 
2, and on each subsequent level of achievement. The data for the two highest 
levels are aggregated, as the percentage of students at the sixth level is rather 
small in most countries. 
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Finland 
OECD average 
Estonia 
Holland 
Norwey 
Poland 
Danmark 
Lichtenstein 
Swiss 
Iceland 
Ireland 
Sweden 
Hungary 
Latvia 
Portugal 
Belgium 
UK 
Germany 
Spain 
France 
Italy 
Slovenia 
Greece 
Slovakia 
Croatia 
Czech Republic 
Lithuania 
Luxembourg 
Russia 
Austria 
Serbia 
Romania 
Bulgaria 
Montenegro 
Albania 



100 


■ Below level 2 ■ Level 2 ■ Level 3 ■ Level 4 ■ Level 5 and 6 


Figure 2: Percentage of students at each proficiency level on the reading scale 
(PISA 2009) - countries are ordered according to the percentage of students 
below PISA level 2 

The data show a big difference in the percentage of students who may be 
considered as functionally illiterate in the reading domain. In Finland, which 
has the highest average achievement, less than 10% of students fall below level 
2 in the reading domain. In OECD countries, an average of 12.6% of students 
remained below level 2 and are consequently treated as functionally illiterate. 
In Estonia, the Netherlands, Norway, Poland and Denmark, less than 15% of 
students can be considered to be functionally illiterate. With this percentage 
of functional illiterates, these countries have already achieved the European 
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benchmarks in education for the year 2020 (less than r5% of students below 
the level 2) (EU, 2009). Many European countries have between 15% and 20% 
functional illiteracy amongst students - Liechtenstein, Switzerland, Iceland, 
Ireland, Sweden, Hungary, Latvia, Portugal, Belgium, Britain, Germany, Spain 
and France. In some other countries, the percentage of functionally illiterate 
students is between 20% and 30%: Italy, Slovenia, Greece, Slovakia, Croatia, the 
Czech Republic, Lithuania, Luxembourg, Russia and Austria. Finally, in Roma¬ 
nia, Bulgaria and Montenegro this percentage exceeds 40%, while in Albania it 
reaches almost 57%. 

Data for Serbia show that about 33% of students have not reached level 
2. This means that every third student aged 15 in Serbia has difficulty in under¬ 
standing complex texts, which can represent a significant obstacle to their fur¬ 
ther education, where reading and understanding textbook texts is an impor¬ 
tant prerequisite to success in school learning. On this basis, it can be assumed 
that a third of students from upper secondary education in Serbia will have sig¬ 
nificant difficulties in continuing their education and finding job opportunities. 

What is the situation with the percentage of students who managed to 
reach level levels 5 and 6, which represent the highest levels of reading literacy? 
In two countries (Finland and Belgium), more than 10% of students attained 
levels 5 and 6, which is the average for OECD countries - 14.5% in Finland and 
11.2% in Belgium. In the following ten countries, the percentage of students at 
the two highest levels is between 7% and 10% (The Netherlands, France, Sweden, 
Iceland, Norway, Switzerland, UK, Germany and Poland), and in twelve coun¬ 
tries it is between 4% and 6% (Hungary, Estonia, Italy, Luxembourg, Greece, the 
Czech Republic, Austria, Portugal, Denmark, Liechtenstein, Slovenia and Slo¬ 
vakia). In other countries, including Serbia, the percentage of students at these 
two levels is below 3%. In Serbia, only about 0.8% of students barely reached 
level 5 in the domain of reading literacy. In other words, if we imagine a school 
with 1,000 students, in most European countries there would be from 40 to 100 
students with the highest level of reading literacy, while in Serbia there would 
only be only 8 students. These data indicate that the education system in Serbia 
does not manage to provide learning opportunities to the best students in order 
to support them in developing reading competence to the highest level. 

In summarising the findings on the distribution of students from Serbia 
at different levels of development of reading literacy, it can be said that after al¬ 
most nine years of compulsory education every third student has failed to reach 
the minimum level of functional literacy (level 2), whereas very few students 
from Serbia manage to reach the highest levels of reading literacy. 

Although every third student from Serbia failed to reach the level of 
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functional literacy in PISA 2009, this result represents a very significant im¬ 
provement when it is compared to findings from the PISA 2006 study. In the 
PISA 2006 study, 52% of the students from Serbia fell below level 2, which 
means that in 2009 the percentage of functionally illiterate students was re¬ 
duced by almost 20 percentage points (Figure 3). 



Figure 3: Change in the percentage of students who failed to reach the level of 
functional literacy in the domain of reading literacy. (PISA 2006 and 2009) 

Based on the data presented in Table 4, it can be seen that the biggest 
changes between the two cycles of the PISA study are related to a decrease in 
the percentage of students falling below level 2. Changes related to the other 
proficiency levels are significantly smaller. This suggests that the education sys¬ 
tem in Serbia significantly improved learning opportunities for most struggling 
students, thus increasing their chances of reaching the level of functional lit¬ 
eracy in the reading domain. 

Linking this finding with the finding that the average achievement of 
Serbian students improved significantly between PISA 2006 and 2009 stud¬ 
ies, it can be concluded that the increase in the average achievement occurred 
primarily as a result of progress made at the lower end of the reading scale. In 
other words, the improvement in average achievement was largely due to the 
fact that the education system improved its capacity to support students who 
are struggling the most. 
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Table 4: Percentage of students in Serbia who have attained certain levels 
of reading literacy. (PISA 2006 and 2009) 


Year 

Below level 2 

Level 2 

Level 3 

Level 4 

Levels 5 and 6 

2006 

51.7 

28.1 

16.0 

3.9 

0.3 

2009 

32.8 

33.2 

25.3 

7.9 

0.8 

Change 

-18.9 

5.1 

9.3 

4.0 

0.5 


Discussion 

The results presented above trigger two issues that should be consid¬ 
ered, and simultaneously provide the basis for formulating certain hypotheses 
as answers to these issues. The first question is: why is the average achievement 
of students in Serbia significantly lower than in most OECD countries and in 
other European countries? Secondly: what could explain the improvement in 
the average achievements made between the 2006 and 2009 PISA studies? 

Assuming that the PISA results mainly reflect students’ experience in 
education, primarily in formal education (facilities, operational procedures, 
typical learning activities, typical patterns of interaction with teachers, text¬ 
books, etc.), rather than differences in so-called “biological potentials,” the an¬ 
swer to the first question should be sought in the education system. We believe 
that when formulating an answer to the first question it is important to bear 
in mind three findings: (a) the average achievement of students from Serbia is 
similar to the average achievements of students from Bulgaria and Romania, 
(b) differences in achievement between students in Serbia are smaller than in 
the other countries, and (c) a small number of students from Serbia reached the 
two highest levels, 5 and 6. 

Countries (Bulgaria and Romania) that have fallen into the same group 
as Serbia for several cycles of PISA (Baucal & Pavlovic Babic, 2010) have three 
important characteristics in common: (a) these are countries in which the over¬ 
all economic situation is worse than in other European countries, and therefore 
the investment in education is substantially lower, which is especially true for 
per capita funding (Eurydice, 2009), (b) educational practice is to a greater ex¬ 
tent knowledge-oriented, i.e., students are mainly supported in the acquisition 
of appropriate academic knowledge and skills rather than in developing key 
competencies (EU, 2002; Eurydice, 2010), and (c) the dominant form of teach¬ 
ing/learning practice in the classroom is lecturing, while active learning, in¬ 
quiry based learning and project learning are rather incidental. (Dimou, 2009; 
Ivic, Pesikan & Antic, 2001; Mintz, 2009; UNICEF, 2001; EU, 2007). 
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Consequently, we believe that the lower average results of students from 
Serbia may be only partially explained by the weaker economic situation. As 
earlier studies (Baucal & Pavlovic Babic, 2010; OECD, 2010) have shown, the 
relationship between the economic situation in a country and investment in 
education, on the one hand, and academic achievements, on the other hand, is 
not strong enough to be the only explanation. We assume that the main expla¬ 
nation for the lower results is related to the fact that teaching and learning in 
Serbian schools is still typically directed toward the appropriation of academic 
knowledge, with traditional lectures being the prevailing form of teaching and 
learning (Dimou, 2009; Ivic, Pesikan & Antic, 2001; UNICEF, 2001). In such 
conditions, there are scarce learning opportunities for the development of the 
key competencies and critical thinking that are typically demanded by PISA 
items, especially those from the highest proficiency levels. This explanation is 
supported by the fact that less than 1% of students from Serbia managed to 
reach levels 5 and 6. 

In this context, another question becomes very intriguing: if teaching in 
schools is predominantly directed towards the acquisition of academic knowl¬ 
edge, and if nothing has changed significantly in this respect in the period 
2006-2009, how can the remarkable achievements in improving the average 
reading scale literacy be explained? The results shown earlier indicate that the 
average achievement was improved largely due to the fact that the percentage 
of students attaining level 2 increased significantly. However, moving students 
from the bottom of the scale across the threshold of the second level was not 
accompanied by an increased number of students at the two highest levels. 

Overall, we believe there is no doubt that students and teachers in PISA 
2009 were more motivated and more engaged than they were in 2006. It is also 
true that the Ministry of Education, which was indifferent towards the PISA 
2006 study, was very supportive and had a feeling of ownership in PISA 2009. 
These changes in attitudes towards the PISA study influenced schools, teachers 
and students to be more motivated to achieve better results. This motivational 
factor contributed immensely to the improvement of achievement at the lower 
end of the scale. Therefore, our assumption is as follows: a significant number of 
students who had difficulty with PISA tasks in 2006 easily gave up solving the 
tasks, while in 2009 these students made an effort to solve at least the tasks at 
the lower PISA levels. However, at the upper end of the scale motivation with¬ 
out adequate competencies could not improve the scores. As a result, in 2009, 
unlike in 2006, many more students from Serbia passed the lower limit of func¬ 
tional literacy, which led to an increase in average achievements but did not 
significantly increase the percentage of Serbian students on the 5 th and 6 th levels. 
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In order to understand the significance of the improvement in achieve¬ 
ment, the general social context in which the students are educated also needs 
to be taken into consideration. Students who participated in the PISA 2009 
study were born in 1993 and started their education in 2000. They are the first 
generation of Serbian students participating in the PISA study who did not 
experience a serious interruption in their education due to a lack of electricity 
or heating, or to strikes, regional conflicts and other events related to the turbu¬ 
lent 1990s. They were educated in relatively stable social conditions after 2000. 
Therefore, it is very likely that the stable social conditions have also contributed 
to a certain extent to the improvement of the academic achievement of Serbian 
students. 

It can be concluded that the relatively low average achievement of Ser¬ 
bian students is very likely the result of the dominance of traditional teaching/ 
learning practices, which encourage the transmission of academic knowledge. 
Furthermore, it seems that the big improvement in the reading domain is not 
primarily a result of an improvement in the quality of education in Serbia; it is 
more likely to be a result of assuring proper conditions for PISA 2009 assess¬ 
ment in Serbia. In other words, PISA 2009 results reveal the actual quality of 
education in Serbia, which was somewhat blurred by the discouraging context 
in the previous PISA 2006 study. Finally, it can be said that PISA 2009 results 
are the first baseline study of the quality of education in Serbia to be used in 
subsequent years for monitoring and policy purposes. 
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